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ABSTRACT 


In the last 30 years, the application of the Delphi technique has been increasing. With the recent availability and es¬ 
tablished popularity of Internet-based research tools, the Internet has been identified as a means for mitigating Delphi 
limitations, maximizing its advantages, and expanding the breadth of its application. The discourse on the application 
of “e-Delphi” has been speculative in nature, however, with pragmatic analysis of Internet-based Delphi administra¬ 
tion being limited in scope. Concomitantly, methodological guidance for conducting the e-Delphi in health education 
is limited, with best-practice implementation strategies yet to emerge. This paper advances the understanding of the 
e-Delphi technique, contributes to the evolution of this Internet-based research methodology, and provides guidance 
for the e-Delphi researcher in health education. Further, it offers a contribution to the discourse about Internet-based 
inquiry generally, and on using the e-Delphi technique in health education specifically. The authors illustrate that 
e-Delphi has a range of benefits that are effective and efficient in assuaging traditional Delphi limitations; neverthe¬ 
less, a set of methodological issues remain unaddressed and make apparent the need for future research investments 
to better understand and alleviate challenges presented for e-Delphi research in health education. 
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INTRODUCTION 

Since its early conception as a ‘galactic 
network’ of globally connected computers, 
the Internet has moved beyond the realm 
of science fiction to become the primary 
global communication medium. Until now, 
no other communication device has been 
so readily adopted with such widespread 
socio-cultural and economic implications. 
While the complexity of the Internet makes 
an accurate measure of its size challenging, 
over 13 billion indexed webpages can be 
identified using common search engines such 
as Yahoo!, Google, Bing and Ask. 1 Nearly four 


out of five people around the world now call 
internet access a basic human right. 2 Health 
information remains one of the most im¬ 
portant topics that Internet users research 
online. Eight out of ten Internet users look 
online for health information, making it the 
third most popular web endeavour (follow¬ 
ing email and accessing search engines). 3 
Furthermore, wireless Internet users are 
more heavily engaged than desktop Internet 
users. Eighty-three percent of wireless Inter¬ 
net users have looked online for health infor¬ 
mation, compared to 70% of Internet users 
who do not use a mobile connection. 3 The 


structure and function of the Internet has 
revolutionized the way that we access, com¬ 
municate and share health information. 
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The Internet revolution has also im¬ 
pacted the manner in which health data is 
collected, transmitted, stored and analysed. 
By way of the World Wide Web, and through 
the interoperability of information systems, 
the Internet has become an important focus 
for scientific inquiry in multiple areas within 
health education and health promotion. For 
example, Jackson 4 and Leung and Lee 5 ex¬ 
plored Internet-based recreation experiences 
and quality of life. Bargh and McKenna, 6 
Engelberg and Sjoberg, 7 Fogel et al. 8 and 
Kraut et al. 9 investigated the psychological 
implications of the Internet as a social venue. 
Quan-Haase and Wellman 10 and Ellison et 
al. 11 considered the impact of the Internet 
on social networking and social support 
for building social capital. Cassell et al., 12 
Eysenbach, 13 Korp, 14 Escoffery et al. 15 and 
Stellefson et al. 16 examined the efficacy of 
‘e-health’ applications for a variety of in¬ 
dividuals and populations. A special issue 
of the journal Health Education Research 
focused exclusively on the digital ‘revolution’ 
occurring within health education research 
and practice (see the Bernhardt and Hubley 17 
editorial for a synopsis). More recently, in 
2010, the American Journal of Health Edu¬ 
cation put out a “call for papers” for social 
media applications in health education. 

Whereas the Internet has been a research 
focus for scholars in health education, the 
Internet can also be used as a research portal 
from which data can be accessed. In the past, 
researchers used the Internet primarily for 
secondary data collection - that is, data cap¬ 
tured from existing Internet-based sources 
including bibliographic databases (e.g., 
peer reviewed journals, library collections, 
etc.), government databases and directories 
(e.g., policy documents, population census, 
etc.), maps, audio-visual archives and data 
archives. 18 New technology-enabled research 
services have emerged which allow health 
education researchers to engage in a variety 
of primary Internet-based data collection 
activities. However, the academy- in general, 
has been slow to respond to the Internet’s 
potential as a research portal or medium 
for data collection purposes). 19,20 In a study 
by Benfield and Szlemko, 21 it was found that 


it was not until the last decade that the use 
of the Internet for primary data collection 
began to increase. Their search of the Web 
of Science® bibliographic database revealed 
only 494 peer reviewed articles published 
between 1996 and 2006 when the keywords 
“Internet research,” “web based research” and 
“electronic data collection” were entered. 

Nevertheless, recent advances in graphics, 
software, real-time interactivity and Internet 
infrastructure continue to reveal new pos¬ 
sibilities for experimental studies and both 
qualitative and quantitative research. 22,23,24 
The use of email or an “e-survey” to collect 
data are now readily accepted tools facilitat¬ 
ing rapid, simple, and inexpensive commu¬ 
nication portals for collecting and managing 
data and participants. 25,26,27,28 Online focus 
groups, e-conferences, and interviews are 
bridging geographical, fiscal, and social gaps 
which previously created chasms between 
researchers and participants. These former 
breaches inevitably led to barriers and 
constraints which presented potentially in¬ 
surmountable research obstacles. 29,30,31 With 
the recent emergence of 3G and 4G coverage 
in smartphone mobile technology, data col¬ 
lection is proliferating, occurring most any¬ 
where in the world, at any time, day or night. 
Smartphones are used by 74.6 million people 
in the U.S., 32 and the penetration rate is even 
higher in Asia and the Pacific regions of the 
world, including China and India. 33 These 
statistics will likely do nothing but increase 
as time and technology progresses. 

Thus, studies conducted O’Connor 
and Madge, 29 Duffy, 23 and Liamputtong 34 
suggest that the discourse on e-research 
needs to be widened to encourage and in¬ 
cubate methodological refinement so that 
Internet-based methodology can evolve, best 
practices can emerge, and guidance can be 
provided for researchers. There is a paucity 
of practice-based guidelines which exist to 
direct Internet-based research activities in 
the health professions. 35 This is of concern 
to the field, given that enhanced awareness 
of the value of e-research and its associated 
challenges is essential to advancing Internet- 
based inquiry. Therefore, through drawing 
on antecedent research - recent reviews, 


select studies and seminal texts, as well as on 
research observations and experiences, this 
paper offers a contribution to the discourse 
about Internet-based inquiry generally, and 
on using the e-Delphi technique in health 
education specifically. It draws on the les¬ 
sons learned from the e-Delphi literature 
and offers commentary based on concrete 
research experience. Finally, advantages and 
limitations of the e-Delphi are suggested, 
along with recommendations for design 
and administration in future health educa¬ 
tion research. 

THE DELPHI TECHNIQUE 

The Delphi technique is used to system¬ 
atically combine expert opinion in order 
to arrive at an informed group consensus 
on a complex problem. 36 It was originally 
developed by the Rand Corporation in the 
1940s as a tool for soliciting opinion from 
a group of experts. The Delphi has evolved 
from its original conceptual and practical 
applications. Landeta 37 reports that the 
Delphi is a legitimate and valuable contribu¬ 
tor to progress on complex social problems 
and issues, while de Meyrick 35 reports that 
the method has a long tradition as a valid 
research technique - especially in the health 
science field. Several studies in health educa¬ 
tion have used the traditional Delphi tech¬ 
nique exclusively to determine consensus 
in a number of important need areas, such 
as matching manuscript submissions with 
appropriate health education journals, 38 
developing research questions 39 and 
determining specific topics for inclusion 
in environmental health curricula. 40 

In principle, the Delphi is a group 
method that is administered by a researcher 
or research team that assembles a panel of 
experts, poses questions, synthesizes feed¬ 
back and guides the group towards com¬ 
mon ground. The Delphi is a method for 
organizing conflicting values and experi¬ 
ences and facilitates the incorporation of 
multiple opinions into consensus. 41,42 This 
is achieved using iterative rounds of sequen¬ 
tial surveys interspersed with controlled 
feedback reports and the interpretation of 
experts’ opinion. Individuals are asked to 
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reconsider their initial positions in light of 
group trends and can make adjustments 
to future survey responses accordingly. 43 
Survey iteration with feedback is continued 
until convergence of opinion reaches some 
point of diminishing returns, which suggests 
agreement on the topic or problem of inter¬ 
est. Research has shown that the empirical 
point of stability with regard to consensus 
for decision making is generally reached 
after the fourth iteration of responses. 44 

Donohoe and Needham 45 claim that the 
Delphi’s popularity may be attributed to 
six basic attributes that distinguish it from 
traditional face-to face group interview 
techniques. The legitimacy and suitability of 
the method for highly-complex problems are 
well-established as are the expert contribu¬ 
tions to the understanding and/or the reso¬ 
lution of important problems. The Delphi is 
particularly well-suited to research problems 
not compatible to linear or precise analytical 
techniques, and where subjective judgement 
on a collective basis could illuminate new 
perspectives. Anonymity encourages partici¬ 
pants to interact free from the constraints 
of personality conflicts or status relations 
thereby reducing the risk for group dynamics 
to negatively influence outcomes. The Delphi 
is defined by flexibility and reflexivity, as it 
allows the researcher to adapt the technique 
to the research context. The researcher can 
structure the data collection tool - the survey, 
so as to collect quantitative or qualitative data 
or both. This allows the researcher consider¬ 
able flexibility in the design phase, but also 
enables the collection of a rich and varied data 
set. Flexibility and reflexivity also allows par¬ 
ticipants to scrupulously think through the 
problem between rounds so that the validity 
of the data and the outcomes are enhanced. 
Unlike other group methods, the Delphi does 
not demand the proximity or face-to-face 
meetings of the researcher with participants. 
The exercise is conducted remotely thereby 
reducing travel costs and the potential for 
group dynamics to manifest. 

THE E-DELPHI 

By its very nature, the Internet offers a 
promising and rapidly evolving medium 
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for Delphi research. The conduct of Delphi 
studies is amenable to the Internet platform 
where iterative collection of data can be 
made more efficient.Originally coined by 
MacEachren et ah, 46 the “e-Delphi” tech¬ 
nique represents recent attempts to com¬ 
puterize the Delphi process so as to optimize 
the method’s ability to organize widespread 
and diverse group thinking, while capital¬ 
izing on the aforementioned methodo¬ 
logical advantages. The e-Delphi relies on 
an Internet-based platform for organizing, 
controlling and facilitating communications 
between the researcher and expert panel. 
When compared with the traxditional pen- 
and-paper approach to data collection, the 
e-Delphi offers unparalleled convenience, 
time and cost savings, and data management 
advantages. Deshpande et al. 47(p - 55) reported 
that the e-Delphi is a “feasible, convenient 
and acceptable alternative to the traditional 
paper-based method.” It is important to 
identify potential research situations where 
the e-Delphi research technique would be 
an especially viable alternative, especially 
as compared to the traditional paper-based 
method. What follows is a description of one 
such relevant research situation. 

There are important conversations on¬ 
going related to the marketing of the health 
education profession for the development 
of the field and the procurement of a sus¬ 
tainable professional base. 48 An important 
prospective research activity for the held 
will likely involve the collection of data 
from prominent experts in health educa¬ 
tion to facilitate consensus in (a) defining 
the purpose and function of the broad, 
international field of health education, and 
(b) developing curriculum standards within 
professional preparation programs for the 
purpose of strengthening the training of 
health education students seeking various 
distinct health-related occupations. Future 
e-Delphi work to establish this consensus 
would ideally capture and share such input 
from distinguished health educators not just 
in the United States, but around the world. 
The e-Delphi study, in and of itself, could 
uniquely and effectively connect the opin¬ 
ions of expert health education researchers 


and practitioners from around the world, 
and provide a research forum to systemati¬ 
cally generate an agreed upon direction for 
the future of the profession. The findings 
from such a technologically-mediated e- 
Delphi study might well mark a watershed 
moment for the field, one which could not 
have been conceptualized or coordinated 
without this kind of research technology. 
Given the potential of the e-Delphi study 
to address these kinds of research questions, 
it is worth distinguishing those numerous 
benefits of the e-Delphi which bear noting. 

Convenience is a benefit for both the 
e-Delphi administrator and the research 
participants. The administrator can log-in to 
the ‘virtual laboratory’ at any time to design, 
adjust, or monitor any number of simultan¬ 
eous activities. For example, the administra¬ 
tor can chose a level of anonymity whereby 
participants may be entirely anonymous 
or they may be revealed to others through 
a user profile (containing information on 
location or expertise). The administrator 
can also control access to information so 
that participants may only be able to access 
their own responses or they may be al¬ 
lowed to see what others have contributed, 
which is integral to facilitating fidelity to 
the methodological process. In contrast to 
the traditional Delphi, e-Delphi research 
participants can access the virtual laboratory 
wherever there is Internet access and when 
it is most convenient for them to do so. The 
process therefore occurs in ‘real-time’, al¬ 
lowing the administrator and participants 
to remain conveniently connected and up 
to date as the e-Delphi progresses. 

Time and cost savings are reported to be 
the most persuasive advantage of Internet- 
based research. 49 Day and Bobeva 50 report 
that the e-Delphi is an increasingly attractive 
alternative for mitigating the traditionally 
long waits between Delphi iterations when 
surface mail is the primary communica¬ 
tion channel. The time associated with the 
conventional Delphi is identified not only 
as a cause of attrition specifically, but also 
a deterrent to Delphi research (for both re¬ 
searchers and participants) generally. In this 
regard, Donohoe and Needham 45 claim that 
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the use of the Internet and the time savings it 
permits enhances the appeal of using Delphi 
methodology and for participating in Del¬ 
phi research. De Villiers et al. 51 report that 
the major advantage of the e-Delphi is the 
cutting of costs, time and effort as it enables 
a group of experts to be canvassed rapidly 
and inexpensively (eliminating printing and 
postage costs) without geographical limita¬ 
tions. They conclude that the technique’s 
suitability for electronic administration 
makes it cost-effective and efficient to use 
for health education research generally and 
for international health education research 
explicitly. This advantageous trait is cor¬ 
roborated by Atkinson and Gold 52 who note 
that the e-Delphi facilitates a cost-efficient 
and practical way to use consensus-building 
strategies without the distance constraints 
imposed by geography. 

Group research usually requires sig¬ 
nificant data management investments. 
Keeping track of personal information and 
managing data collection and analysis are 
common challenges. 49 In a survey of quali¬ 
tative researchers, Miles and Huberman 53 
reported that 75% are using electronic data 
management software, while Mann and 
Stewart 49 reported an increased use because 
new software offers several advantages. It 
facilitates data management, reduces risk 
of error and makes the analytic process 
more transparent. In the burgeoning era 
of cloud computing, researchers in health 
education should strongly consider the 
important security and access benefits that 
research endeavours such as the e-Delphi 
enable. Such benefits to data management 
in research are sure to proliferate with time; 
therefore, health educators should consider¬ 
ing adopting these Web-based software and 
storage technologies when designing and 
conducting e-Delphi studies. 

In light of these evident advantages, a 
literature search was conducted to determine 
the extent of static inquiry done using the 
Internet to conduct Delphi research. A Web 
of Science® bibliographic database search 
using the keywords “Delphi” and “Internet” 
produced 79 peer reviewed articles pub¬ 
lished between 1988 and 2010. Of these, 


11 were unrelated because they focused on 
a computer software program of the same 
name - Delphi. Of the remaining 68 articles, 
20 applied a traditional Delphi to the study 
of the Internet 54,55,56 and 46 reported the re¬ 
sults of an e-Delphi study (all published after 
1998). In the 46 articles where the Internet 
was used to support, facilitate, or enable 
Delphi inquiry, the primary Internet func¬ 
tion was as an email portal through which 
surveys were delivered and responses were 
collected. 57,58,59 Two articles report using web 
pages for posting and sharing Delphi reports 
and resources with panellists, three report 
using online conference technology for panel 
communication, and twenty use electronic 
survey software (e-survey) for collecting, 
managing and analysing data 47,60 ' 64 

Figure 1 depicts the linear progression 
of e-Delphi publications in peer-reviewed 
scientific journals over the last 10+ years. It is 
important to remember that while the total 
number of e-Delphi’s is small compared to 
the number of traditional Delphi’s published 
in the scientific literature during the same 
time period, the number of researchers us¬ 
ing the Internet for Delphi administration 
has been steadily increasing since 1997. The 
growing interest in reporting e-Delphi (ver¬ 


sus traditional Delphi) results is evidence of 
its benefits. Importantly, however, only five 
studies have contributed any methodological 
critique 65,66 of the e-Delphi. This illustrates 
that manuscripts on e-Delphi design and 
administration processes are consistently 
missing from the literature. 

e-Delphi Limitations 

Donohoe and Needham 45 recognize 
that using the Internet for Delphi research 
presents a new and exciting research fron¬ 
tier, but they caution that deficiencies in 
methodological discourse is putting at risk 
the design, implementation, success and 
evolution of e-Delphi research. Clearly, this 
is an important and evolving contribution 
area for Delphi research specifically, and 
Internet-based research generally. Despite 
the reported advantages of the e-Delphi, 
the review of the literature described above 
suggests that methodological limitations 
may be manifesting and precluding wide¬ 
spread implementation of the research 
technique. Internet accessibility challenges, 
technological difficulties and the incon¬ 
venience of entering data into computer- 
based data screens over the convenience of 
hard copy ‘page-flipping’ (note: this is also 
reported as an advantage) are presenting 
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new and unforeseen challenges. While 
many of the issues can be considered char¬ 
acteristic of computers and the Internet, 
problems associated with traditional meth¬ 
ods can be intensified by the conditions 
of the virtual landscape. These include 
perceived anonymity, respondent identity 
(real or perceived) and data accuracy (re¬ 
sponse selection control and transmission 
errors). 67 ’ 68 ’ 69 Because of the problems 
associated with e-research, select issues — 
access and control — are brought forward 
for discussion specifically as they pertain to 
e-Delphi research in health education. 

Access 

The literature establishes that the In¬ 
ternet is a tool for extending the potential 
research population or sample but it is also 
assumes that people have access to the In¬ 
ternet. 49 The United Nations International 
Telecommunications Union (ITU) reported 
that the number of Internet users surpassed 
2 billion in 2010 and projected that half of 
the world’s population will have broadband 
access by 2015. 70 While the United States has 
historically maintained the most users on a 
per country basis, growth is not limited to 
developed nations. In fact, the spatial reach 
or “penetration rate” of the Internet contin¬ 
ues to grow within developing countries as 
well as increasingly within remote corners of 
the world. China, for example, surpassed the 
United States in 2006 with the most Internet 
users - 160 million, as a result of their rap¬ 
idly improving infrastructure and growing 
economy. 70 Economic and infrastructure 
growth, however, has failed to thwart slow 
Internet penetration rates and access con¬ 
straints in remote, rural and developing 
areas. Bernhardt and Hubley 17 caution that 
despite the promise and potential, many 
significant obstacles remain to be overcome 
in health education research including, but 
not limited to, the vast divide that exists 
between those with access to Internet tech¬ 
nologies and those who are unlikely to ever 
have such access. 

Beyond infrastructure, unreliable In¬ 
ternet access can also present a significant 
challenge for e-Delphi administrators and 
participants. Where ‘dial-up’ access is the 
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norm or where access is interrupted by 
weather, controlled by political interven¬ 
tions, or terminated by acts of war or ter¬ 
rorism, the e-Delphi can be significantly 
affected. The challenge of sustaining elec¬ 
tronic communications with participants 
is magnified by these constraints - par¬ 
ticularly when conducting health research 
in rural, Indigenous, or impoverished 
communities. Access to mobile networks is 
becoming more readily available, however, 
with coverage in over 90% of the world’s 
population and 143 countries offering 3G 
services. 70 Unfortunately, Internet access 
remains expensive, especially in low-income 
developing countries such as several Latin 
America nations and Africa. 70 Nevertheless, 
studies are now being done to develop and 
refine open source, standards-based tools for 
mobile data collection, aggregation, analysis 
and reporting in these types of low income 
regions of the world. 73 

To maximize communications and 
time-savings as well as to mitigate these 
potential limitations, it is recommended 
that the researcher offer a pen-and-paper 
survey alternative, invest in developing 
clear directions for survey completion, and 
pilot test Internet-based surveys and com¬ 
munications (to avoid simple technological 
errors). It is also highly recommended that 
the e-Delphi administrator make themselves 
or an assistant available for technical support 
throughout the duration of a Delphi exer¬ 
cise (many e-survey providers offer phone 
and Internet support). To enable research 
networks using the Internet, Bernhardt 
and Hubley 17 and Donohoe 18 constructed 
websites for sharing research information 
(a central site from which all information, 
surveys and resources could be accessed), 
used a browser-friendly survey delivery 
system, pilot-tested the surveys and delivery 
mechanisms, and provided technological 
support (for participants) to mitigate access 
issues. They report that these research design 
and administration decisions facilitated a 
generally seamless e-Delphi process. 

Research Control 

It has been suggested that control of 
the ‘virtual laboratory’ is more difficult 


than the traditional research laboratory. 
Concerns related to false representation, 
lack of verbal interaction, and participant 
distractions have been reported in the 
Internet inquiry literature. 49,74 In the case 
of the e-Delphi, the research environment 
is ‘virtual’ regardless of the medium of 
communication, with verbal and physical 
interaction not acting as a distinguishing 
feature of the method. However, control 
issues can be magnified when using the e- 
Delphi over the traditional Delphi method 
(where the Delphi survey is delivered by 
post to the expert respondent). 

The Delphi method, by its very nature, 
is highly selective about the type of re¬ 
spondents that are invited to participate, 
and this is also true in health education 
research. 51,52 The Delphi relies on expert 
opinion and this is a distinguishing feature 
and methodological absolute. While the 
literature details a number of ways in which 
expertise can be assessed and/or established 
(e.g. self-identification, screening survey, 
third party verification 45 ), the anonymity 
of the Internet presents concerns related 
to representation. In effect, when using the 
Internet to conduct research, “there is a high 
degree of uncertainty... in terms of‘know¬ 
ing’ the identity of the other” and this may 
raise concerns when seeking Institutional 
Review Board (IRB) approval for e-Delphi 
research. 75 (p 5) To mitigate potential control 
limitations and to address potential IRB 
concerns, it is recommended that a secure 
hyperlink be provided to expert participants 
(unique and accessed only by each individual 
participant - so as not to be forwarded or 
opened by others) along with passwords 
(in a separate email preferably) to access 
the e-surveys (see Table 1). Bernhardt and 
Hubley 17 report that this practice provides 
reassurance that the survey respondent 
was indeed the invited expert participant. 
However, this method is not foolproof and 
control remains an issue yet unresolved in 
Internet-based inquiry. Further research 
must be conducted to better understand 
how to control the virtual laboratory and 
avoid representation concerns. By extension, 
researchers are encouraged to share their 
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Table 1. e-Delphi Limitations and Mitigation Measures 

Limitations 

Mitigation Measures 

Internet Access and 
Technology 

■ Make traditional pen-and-paper survey available to those without access or to those who 
experience difficulties 

■ Maintain copies of all documents, data, and reports in case of hardware failure 

■ Offer financial assistance for those without access or without reliable access (cost to use Inter¬ 
net service at library, cafe or other) 

■ Select an established Internet-based survey provider (consult the literature for guidance) with 
help features for survey designers and respondents 

* Downloadable guides and' real-time chat' are particularly valuable 

■ Provide clear instructions to participants and consider developing a 'frequently asked ques¬ 
tions' or 'how-to guide' 

■ Provide technical support through a variety of mediums (e.g., email, phone, Skype) for the 
duration of the exercise 

■ Pilot test all communications and surveys to avoid interpretation and technological difficulties, 
then test again 

Experimental Control 

■ Identify known or possible distractions and time administration threats accordingly. E.g. avoid 
vacation periods, major conferences, or 'high-season' times 

■ Invest in strategies to avoid misrepresentation 

■ Create a screening tool to capture and winnow out the most appropriate experts for the 
study 

■ Corroborate self-identified experts through membership organizations, academic or profes¬ 
sional institutions, publications, etc. 

■ Ensure secure survey access (unique passwords and hyperlinks) 


e-Delphi methodological experiences with 
the health education research community 
so that best-practices can evolve. 

On the basis of the advantages and 
limitations discussed here, Table 1 intro¬ 
duces a set of mitigation measures to assist 
e-Delphi architects and administrators in 
overcoming Internet access and experimen¬ 
tal control impediments. 

Maximizing the Potential of the e-Delphi 
Technique 

Table 2 presents a set of recommenda¬ 
tions to address the most pressing limita¬ 
tions of the traditional Delphi by capitalizing 
on the potential advantages of Internet- 
based tools and techniques. 

When using the Delphi technique in 
health education, it is recommended that 
the researcher critically assess the advan¬ 
tages and limitations of Internet-based 
tools before making experimental design 
decisions. Should an e-Delphi be selected 
as the methodological best-fit for the study, 
an Internet-based research portal should be 


established to legitimize the study as well as 
to serve as the primary communication me¬ 
dium between the researcher(s) and research 
participants. Information about the study 
(purpose, objectives, timelines, ethics, etc.), 
contact information, password protected e- 
survey links and Delphi round reports can be 
posted to ensure transparency throughout 
the research process. Before selecting an 
e-survey software or service provider (e.g., 
Qualtrics, SurveyMonkey), several opera¬ 
tional factors should be considered, such as: 
design features and interface (i.e., ease of use, 
accessibility, etc.), level of respondent access 
and technological support, and options 
for data management and analysis. Many 
services now offer an upgrade option that 
provides access to additional data storage 
as well as advanced data analysis tools (e.g. 
response tracking, attrition monitoring, 
and statistical reporting). Throughout the 
e-Delphi process, it is also highly recom¬ 
mended that the researchers maintain up- 
to-date communications with the research 


participants. This can be accomplished 
through email and through regular updates 
to the Internet-based research portal - so 
as to mitigate attrition and to keep on track 
with established timelines. 

Additionally, the use of mobile and smart 
phone technology to collect data during 
e-Delphi studies is an important area to 
investigate in the literature. Mobile Internet 
usage is growing rapidly worldwide, making 
real-time information tools more readily 
available to clinicians, practitioners and the 
public. 76 Early efforts made to tap the power 
of mobile software tools to engage research 
participants in the data collection process 
will likely represent an important step in 
improving health education research. With 
greater adoption of mobile and smartphone 
technology for data collection, there will 
likely be a set of new methodological e- 
Delphi issues to consider, particularly related 
to the previously cited limitations of access 
to the Internet, control over participant 
access to survey instruments, and other 
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Table 2. Traditional Delphi Limitations and e-Delphi Recommendations 

Limitations 

Recommendations 

Design Sensitivity 

■ Critically assess the advantages and limitations of Internet-based tools before making design decisions 
* Consult Delphi and Internet-based research literature for guidance 

Recruitment 

■ Make use of Internet-based databases and communications to identify and recruit potential participants 
* Establish firm timelines and make them clear to potential participants 

■ Establish a research portal to share information to legitimize the Delphi study 

Time Commitment 

* Use an Internet-based survey platform to reduce time required for postal surveys and communications 
■ Set firm deadlines, provide deadline reminders and adhere to them 

* Consider investing in Internet-based survey platform upgrades to enable advanced information sharing 

* Make use of Internet-based survey platform data processing tools such as response tracking and 
statistical reporting 

Attrition 

* Select participants with high interest in the research problem and/or results 

■ Inform participants of the process and goals at the outset 

■ Invest in transparency - share information and results on research web portal 

* Maintain communications with participants throughout exercise and provide regular timeline reminders 

* Monitor attrition through Internet-based survey platform 

Consensus 

■ Different rules of thumb are considered equally valid - consult the literature for guidance (standard 
deviation is commonly used to access consensus) 

* Construct Internet-based survey tool to include a consensus monitoring function 

* Lack of consensus must be considered a valuable finding 


related consumer health informatics issues 
impacting mobile device usability. Future 
research should investigate how access to 
the Internet on mobile devices impacts the 
e-Delphi with regards to access and control 
over the research process. 

CONCLUSION 

Internet-based technologies are with¬ 
out a doubt, transforming research. The 
traditional grounds and audiences for 
research are shifting; 67 innovative data 
capture and knowledge sharing methods 
are developing; 46 space-time relations and 
geographical boundaries are changing; 77 
and the kinds of research issues that are 
being explored are evolving. 68 Whether the 
Internet is being used as a portal for health 
education or as a channel for conducting 
health education research, it is clear that the 
Internet can have a significant impact on the 
ways in which health education research and 
practice are conducted in the future. 17 

In particular, the e-Delphi offers unpar¬ 


alleled convenience, time and cost savings, 
and data management options. Although 
the e-Delphi technique is a creative and 
efficient method for facilitating health edu¬ 
cation research, there are limitations associ¬ 
ated with the Delphi study in this relatively 
new “virtual landscape.” While many of the 
issues can be considered to be a reflection 
of the nature of technology and the Inter¬ 
net, some limitations associated with the 
traditional Delphi can be intensified and 
new issues can present when the Internet is 
used to enable the technique. It must also be 
acknowledged that the e-Delphi and other 
Internet-based approaches are simply tools 
to be used and applied, when appropriate, 
within the context of health education 
theories, values, principles and models. At 
this time, additional research is needed to 
evaluate the effects of Internet-based re¬ 
search tools such as the e-Delphi for health 
education and to explore its effects relative to 
traditional educational approaches. None¬ 
theless, by adhering to the suggestions for 


overcoming stated methodological obstacles, 
the health education researcher can make 
the most of this most promising e-research 
technique. In doing so, researchers in health 
education can make strong contributions 
to the evolution of methodological best- 
practices for this most relevant, consensus¬ 
building technique. 
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